Towards a common framework for multimodal generation: The behavior markup language

Kopp S, Krenn B, Marsella S, et al. Towards a common framework for multimodal generation: The behavior markup language. In: Gratch J, ed. Procedings Intelligent Virtual Agents 2006. LNAI. Vol 4133. Berlin: Springer; 2006: 205-217.This paper describes an international effort to unify a multimodal behavior generation framework for Embodied Conversational Agents (ECAs). We propose a three stage model we call SAIBA where the stages represent intent planning, behavior planning and behavior realization. A Function Markup Language (FML), describing intent without referring to physical behavior, mediates between the first two stages and a Behavior Markup Language (BML) describing desired physical realization, mediates between the last two stages. In this paper we will focus on BML. The hope is that this abstraction and modularization will help ECA researchers pool their resources to build more sophisticated virtual humans

Teaching computers to conduct spoken interviews: Breaking the realtime barrier with learning

Abstract. Several challenges remain in the effort to build software capable of conducting realtime dialogue with people. Part of the problem has been a lack of realtime flexibility, especially with regards to turntaking. We have built a system that can adapt its turntaking behavior in natural dialogue, learning to minimize unwanted interruptions and “awkward silences”. The system learns this dynamically during the interaction in less than 30 turns, without special training sessions. Here we describe the system and its performance when interacting with people in the role of an interviewer. A prior evaluation of the system included 10 interactions with a single artificial agent (a non-learning version of itself); the new data consists of 10 interaction sessions with 10 different humans. Results show performance to be close to a human’s in natural, polite dialogue, with 20 % of the turn transitions taking place in under 300 msecs and 60 % under 500 msecs. The system works in real-world settings, achieving robust learning in spite of noisy data. The modularity of the architecture gives it significant potential for extensions beyond the interview scenario described here

On Attention Mechanisms for AGI Architectures: A Design Proposal

Abstract. Many existing AGI architectures are based on the assumption of infinite computational resources, as researchers ignore the fact that real-world tasks have time limits, and managing these is a key part of the role of intelligence. In the domain of intelligent systems the management of system resources is typically called “attention”. Attention mechanisms are necessary because all moderately complex environments are likely to be the source of vastly more information than could be processed in realtime by an intelligence’s available cognitive resources. Even if sufficient resources were available, attention could help make better use of them. We argue that attentional mechanisms are not only nice to have, for AGI architectures they are an absolute necessity. We examine ideas and concepts from cognitive psychology for creating intelligent resource management mechanisms and how these can be applied to engineered systems. We present a design for a general attention mechanism intended for implementation in AGI architectures

Predictive Heuristics for Decision-Making in Real-World Environments

Abstract. In this paper we consider the issue of endowing an AGI system with decision-making capabilities for operation in real-world environments or those of comparable complexity. While action-selection is a critical function of any AGI system operating in the real-world, very few applicable theories or methodologies exist to support such functionality, when all necessary factors are taken into account. Decision theory and standard search techniques require several debilitating simplifications, including determinism, discrete state spaces, exhaustive evaluation of all possible future actions and a coarse grained representation of time. Due to the stochastic and continuous nature of real-world environments and inherent time-constraints, direct application of decision-making methodologies from traditional decision theory and search is not a viable option. We present predictive heuristics as a way to bridge the gap between the simplifications of decision theory and search, and the complexity of real-world environments

Affordances and Cognitive Walkthrough for Analyzing Human-Virtual Human Interaction

This study investigates how the psychological notion of affordance, known from human computer interface design, can be adopted for the analysis\ud and design of communication of a user with a Virtual Human (VH), as a novel interface. We take as starting point the original notion of affordance, used to\ud describe the function of objects for humans. Then, we dwell on the humancomputer interaction case when the object used by the human is (a piece of software in) the computer. In the next step, we look at human-human\ud communication and identify actual and perceived affordances of the human\ud body and mind. Then using the generic framework of affordances, we explain\ud certain essential phenomena of human-human multimodal communication. Finally, we show how they carry over to the case of communicating with a 'designed human', that is an VH, whose human-like communication means may\ud be augmented with ones reminiscent of the computer and fictive worlds. In the\ud closing section we discuss and reformulate the method of cognitive walkthrough to make it applicable for evaluating the design of verbal and nonverbal\ud interactive behaviour of VHs

Perceptual grouping

Modifying a digital sketch may require multiple selections before a particular editing tool can be applied. Especially on large interactive surfaces, such interactions can be fatiguing. Accordingly, we propose a method, called Suggero, to facilitate the selection process of digital ink. Suggero identifies groups of perceptually related drawing objects. These "perceptual groups" are used to suggest possible extensions in response to a person's initial selection. Two studies were conducted. First, a background study investigated participant's expectations of such a selection assistance tool. Then, an empirical study compared the effectiveness of Suggero with an existing manual technique. The results revealed that Suggero required fewer pen interactions and less pen movement, suggesting that Suggero minimizes fatigue during digital sketching.Ye

Simulation and Anticipation as Tools for Coordinating with the Future

A key goal in designing an artificial intelligence capable of performing complex tasks is a mechanism that allows it to efficiently choose appropriate and relevant actions in a variety of situations and contexts. Nowhere is this more obvious than in the case of building a general intelligence, where the contextual choice and application of actions must be done in the presence of large numbers of alternatives, both subtly and obviously distinct from each other. We present a framework for action selection based on the concurrent activity of multiple forward and inverse models. A key characteristic of the proposed system is the use of simulation to choose an action: the system continuously simulates the external states of the world (proximal and distal) by internally emulating the activity of its sensors, adopting the same decision process as if it were actually operating in the world, and basing subsequent choice of action on the outcome of such simulations. The work is part of our larger effort to create new observation-based machine learning techniques. We describe our approach, an early implementation, and an evaluation in a classical AI problem-solving domain: the Sokoban puzzle

Mutually coordinated anticipatory multimodal interaction

We introduce our research on anticipatory and coordinated interaction between a virtual human and a human partner. Rather than adhering to the turn taking paradigm, we choose to investigate interaction where there is simultaneous expressive behavior by the human interlocutor and a humanoid. We have designed various applications in which we can study and specify such behavior, in particular behavior that requires synchronization based on predictions from performance and perception. We have some preliminary observations – based on the literature and in analogy with our applications – on the role of predictions in conversations. Architectural consequences for the design of virtual humans are presented